{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "084u8u0DpBlo"
      },
      "source": [
        "# Gemini API: Prompting with Video"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Video.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ],
      "metadata": {
        "id": "wnQ_LVlzIeXo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook provides a quick example of how to prompt Gemini 1.5 Pro using a video file. In this case, you'll use a short clip of [Big Buck Bunny](https://peach.blender.org/about/)."
      ],
      "metadata": {
        "id": "q7QvXQMrIhuZ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qLuL9m7KhvxR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50052ade-759a-437e-8476-4b8a1344c4c8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install -U -q google-generativeai"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import google.generativeai as genai"
      ],
      "metadata": {
        "id": "ATIbQM0NHhkj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Authentication Overview\n",
        "\n",
        "**Important:** The File API uses API keys for authentication and access. Uploaded files are associated with the API key's cloud project. Unlike other Gemini APIs that use API keys, your API key also grants access data you've uploaded to the File API, so take extra care in keeping your API key secure. For best practices on securing API keys, refer to Google's [documentation](https://support.google.com/googleapi/answer/6310037)."
      ],
      "metadata": {
        "id": "ngyzKAu3Nw5k"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l8g4hTRotheH"
      },
      "source": [
        "### Setup your API key\n",
        "\n",
        "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d6lYXRcjthKV"
      },
      "outputs": [],
      "source": [
        "from google.colab import userdata\n",
        "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n",
        "genai.configure(api_key=GOOGLE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MNvhBdoDFnTC"
      },
      "source": [
        "## Extract frames\n",
        "\n",
        "The Gemini API currently does not support video files directly. Instead, you can provide a series of timestamps and image files.\n",
        "\n",
        "We will extract 1 frame a second from a the short film \"Big Buck Bunny\" file using [OpenCV](https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html).\n",
        "\n",
        "> \"Big Buck Bunny\" is (c) copyright 2008, Blender Foundation / www.bigbuckbunny.org and [licensed](https://peach.blender.org/about/) under the [Creative Commons Attribution 3.0](http://creativecommons.org/licenses/by/3.0/) License.\n",
        "\n",
        "Note: You can also [upload your own files](https://github.com/google-gemini/cookbook/tree/main/examples/Upload_files.ipynb) to use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_HzrDdp2Q1Cu"
      },
      "outputs": [],
      "source": [
        "video_file_name = \"https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_2KHEawhDD7"
      },
      "source": [
        "Use OpenCV to extract image frames from the video at 1 frame per second."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jW9ilYCdL99M",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9583f8c3-65b5-4740-f690-bf2c1bda6176"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4 at 1 frame per second. This might take a bit...\n",
            "Completed video frame extraction!\n",
            "\n",
            "Extracted: 597 frames\n"
          ]
        }
      ],
      "source": [
        "import cv2\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "# Create or cleanup existing extracted image frames directory.\n",
        "FRAME_EXTRACTION_DIRECTORY = \"/content/frames\"\n",
        "FRAME_PREFIX = \"_frame\"\n",
        "def create_frame_output_dir(output_dir):\n",
        "  if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "  else:\n",
        "    shutil.rmtree(output_dir)\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "def extract_frame_from_video(video_file_path):\n",
        "  print(f\"Extracting {video_file_path} at 1 frame per second. This might take a bit...\")\n",
        "  create_frame_output_dir(FRAME_EXTRACTION_DIRECTORY)\n",
        "  vidcap = cv2.VideoCapture(video_file_path)\n",
        "  fps = vidcap.get(cv2.CAP_PROP_FPS)\n",
        "  frame_duration = 1 / fps  # Time interval between frames (in seconds)\n",
        "  output_file_prefix = os.path.basename(video_file_path).replace('.', '_')\n",
        "  frame_count = 0\n",
        "  count = 0\n",
        "  while vidcap.isOpened():\n",
        "      success, frame = vidcap.read()\n",
        "      if not success: # End of video\n",
        "          break\n",
        "      if int(count / fps) == frame_count: # Extract a frame every second\n",
        "          min = frame_count // 60\n",
        "          sec = frame_count % 60\n",
        "          time_string = f\"{min:02d}:{sec:02d}\"\n",
        "          image_name = f\"{output_file_prefix}{FRAME_PREFIX}{time_string}.jpg\"\n",
        "          output_filename = os.path.join(FRAME_EXTRACTION_DIRECTORY, image_name)\n",
        "          cv2.imwrite(output_filename, frame)\n",
        "          frame_count += 1\n",
        "      count += 1\n",
        "  vidcap.release() # Release the capture object\\n\",\n",
        "  print(f\"Completed video frame extraction!\\n\\nExtracted: {frame_count} frames\")\n",
        "\n",
        "extract_frame_from_video(video_file_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-z4zsCUlaru"
      },
      "source": [
        "## Upload frames using the File API\n",
        "\n",
        "Once we have the frames extracted, we are ready to upload the frames to the API.\n",
        "\n",
        "The File API accepts files under 2GB in size and can store up to 20GB of files per project. Files last for 2 days and cannot be downloaded from the API.\n",
        "\n",
        "We will just upload 10 frames so this example runs quickly. You can modify the code below to upload the entire video."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JSd4s0YygV9r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45f5b7b6-43e5-4e37-f069-428355089bda"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Uploading 10 files. This might take a bit...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:40.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:41.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:42.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:43.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:44.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:45.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:46.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:47.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:48.jpg...\n",
            "Uploading: /content/frames/BigBuckBunny_320x180_mp4_frame00:49.jpg...\n",
            "Completed file uploads!\n",
            "\n",
            "Uploaded: 10 files\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "class File:\n",
        "  def __init__(self, file_path: str, display_name: str = None):\n",
        "    self.file_path = file_path\n",
        "    if display_name:\n",
        "      self.display_name = display_name\n",
        "    self.timestamp = get_timestamp(file_path)\n",
        "\n",
        "  def set_file_response(self, response):\n",
        "    self.response = response\n",
        "\n",
        "def get_timestamp(filename):\n",
        "  \"\"\"Extracts the frame count (as an integer) from a filename with the format\n",
        "     'output_file_prefix_frame00:00.jpg'.\n",
        "  \"\"\"\n",
        "  parts = filename.split(FRAME_PREFIX)\n",
        "  if len(parts) != 2:\n",
        "      return None  # Indicates the filename might be incorrectly formatted\n",
        "  return parts[1].split('.')[0]\n",
        "\n",
        "# Process each frame in the output directory\n",
        "files = os.listdir(FRAME_EXTRACTION_DIRECTORY)\n",
        "files = sorted(files)\n",
        "files_to_upload = []\n",
        "for file in files:\n",
        "  files_to_upload.append(\n",
        "      File(file_path=os.path.join(FRAME_EXTRACTION_DIRECTORY, file)))\n",
        "\n",
        "# Upload the files to the API\n",
        "# Only upload a 10 second slice of files to reduce upload time.\n",
        "# Change full_video to True to upload the whole video.\n",
        "full_video = False\n",
        "\n",
        "uploaded_files = []\n",
        "print(f'Uploading {len(files_to_upload) if full_video else 10} files. This might take a bit...')\n",
        "\n",
        "for file in files_to_upload if full_video else files_to_upload[40:50]:\n",
        "  print(f'Uploading: {file.file_path}...')\n",
        "  response = genai.upload_file(path=file.file_path)\n",
        "  file.set_file_response(response)\n",
        "  uploaded_files.append(file)\n",
        "\n",
        "print(f\"Completed file uploads!\\n\\nUploaded: {len(uploaded_files)} files\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oOZmTUb4FWOa"
      },
      "source": [
        "## List Files\n",
        "\n",
        "After uploading the file, you can verify the API has successfully received the files by calling `files.list`.\n",
        "\n",
        "`files.list` lets you see all files that have been uploaded to the File API that are associated with the Cloud project your API key belongs to. Only the `name` (and by extension, the `uri`) are unique."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SHMVCWHkFhJW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        },
        "outputId": "c4f0d4d4-43e8-49ca-bd6d-c72cf0ceb731"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://generativelanguage.googleapis.com/v1beta/files/q6y8iak1ky99\n",
            "https://generativelanguage.googleapis.com/v1beta/files/iilxn5wjze28\n",
            "https://generativelanguage.googleapis.com/v1beta/files/icbgp7xsxhe3\n",
            "https://generativelanguage.googleapis.com/v1beta/files/5gmr6rkakb13\n",
            "https://generativelanguage.googleapis.com/v1beta/files/f1hh0z86bg72\n",
            "https://generativelanguage.googleapis.com/v1beta/files/wt6zcf2sr5f6\n",
            "https://generativelanguage.googleapis.com/v1beta/files/ax2qg09qlt8z\n",
            "https://generativelanguage.googleapis.com/v1beta/files/8myngurprfsd\n",
            "https://generativelanguage.googleapis.com/v1beta/files/5ygt6dcc0kn4\n",
            "https://generativelanguage.googleapis.com/v1beta/files/chfsbddbbn7j\n"
          ]
        }
      ],
      "source": [
        "# List files uploaded in the API\n",
        "for n, f in zip(range(len(uploaded_files)), genai.list_files()):\n",
        "  print(f.uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPPOECHzsIGJ"
      },
      "source": [
        "## Generate Content\n",
        "\n",
        "After the file has been uploaded, you can make `GenerateContent` requests that reference the File API URI.\n",
        "\n",
        "To understand videos with Gemini 1.5 Pro, provide 2 consecutive `Part`s for each frame: a `text` part with the **timestamp** and `fileData` part with the frame's **image URI**:\n",
        "\n",
        "```\n",
        "part { text = \"00:00\" }\n",
        "part { fileData = fileData {\n",
        "  fileUri = \"https://generativelanguage.googleapis.com/v1/files/frame-0\"\n",
        "  mimeType = \"image/jpeg\"\n",
        "}}\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZYVFqmLkl5nE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "e764f50f-9f28-4e4f-a558-db021fb759cd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The video shows a large, grumpy rabbit emerging from its burrow. The burrow is surrounded by lush green grass and rocks. The rabbit has a grumpy expression on its face. The background of the video is a beautiful landscape with mountains and trees.\n"
          ]
        }
      ],
      "source": [
        "# Create the prompt.\n",
        "prompt = \"Describe this video.\"\n",
        "\n",
        "# Set the model to Gemini 1.5 Pro.\n",
        "model = genai.GenerativeModel(model_name=\"models/gemini-1.5-pro-latest\")\n",
        "\n",
        "# Make GenerateContent request with the structure described above.\n",
        "def make_request(prompt, files):\n",
        "  request = [prompt]\n",
        "  for file in files:\n",
        "    request.append(file.timestamp)\n",
        "    request.append(file.response)\n",
        "  return request\n",
        "\n",
        "# Make the LLM request.\n",
        "request = make_request(prompt, uploaded_files)\n",
        "response = model.generate_content(request,\n",
        "                                  request_options={\"timeout\": 600})\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IrPDYdQSKTg4"
      },
      "source": [
        "## Delete Files\n",
        "\n",
        "Files are automatically deleted after 2 days or you can manually delete them using `files.delete()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d4eO8ZXoKdZf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc4d2de1-563b-4cdf-9e4d-c22d584429d1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Deleting 10 images. This might take a bit...\n",
            "files/ax2qg09qlt8z\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:43.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/ax2qg09qlt8z\n",
            "files/wt6zcf2sr5f6\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:44.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/wt6zcf2sr5f6\n",
            "files/f1hh0z86bg72\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:45.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/f1hh0z86bg72\n",
            "files/5gmr6rkakb13\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:46.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/5gmr6rkakb13\n",
            "files/icbgp7xsxhe3\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:47.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/icbgp7xsxhe3\n",
            "files/iilxn5wjze28\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:48.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/iilxn5wjze28\n",
            "files/q6y8iak1ky99\n",
            "Deleted /content/frames/BigBuckBunny_320x180_mp4_frame00:49.jpg at URI https://generativelanguage.googleapis.com/v1beta/files/q6y8iak1ky99\n",
            "Completed deleting files!\n",
            "\n",
            "Deleted: 10 files\n"
          ]
        }
      ],
      "source": [
        "print(f'Deleting {len(uploaded_files)} images. This might take a bit...')\n",
        "for file in uploaded_files:\n",
        "  genai.delete_file(file.response.name)\n",
        "  print(f'Deleted {file.file_path} at URI {file.response.uri}')\n",
        "print(f\"Completed deleting files!\\n\\nDeleted: {len(uploaded_files)} files\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Learning more\n",
        "\n",
        "The File API lets you upload a variety of multimodal MIME types, including images and audio formats. The File API handles inputs that can be used to generate content with [`model.generateContent`](https://ai.google.dev/api/rest/v1/models/generateContent) or [`model.streamGenerateContent`](https://ai.google.dev/api/rest/v1/models/streamGenerateContent).\n",
        "\n",
        "The File API accepts files under 2GB in size and can store up to 20GB of files per project. Files last for 2 days and cannot be downloaded from the API.\n",
        "\n",
        "* Learn more about the [File API](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_API.ipynb) with the quickstart.\n",
        "\n",
        "* Learn more about prompting with [media files](https://ai.google.dev/tutorials/prompting_with_media) in the docs, including the supported formats and maximum length."
      ],
      "metadata": {
        "id": "K5oUCqb6IUnH"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}